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Rank order data have features which sometimes make them more useful than data based on pre- 
defined rating scales (Unacre 1990), but the utility of such data has been severely restricted by 
the lack of convenient and informative analytical techniques. A ranked comparison, whether of 
a pair of objects or of larger groupings, produces ordinal counts not interval measures. 
Consequently, to analyze rankings using techniques designed for interval data necessarily distorts 
and confuses their meaning. Though there are analytical techniques intended precisely for these 
types of data (Bradley & Terry 1952, David 1988, Critchlow 1985), they tend to be inaccessible 
and demanding on the analyst 

In this paper, three case studies are presented, demonstrating the application of straight-forward 
Rasch techniques to these types of data. 



Paired Comparisons: Forreri Qjnira 

Paired comparisons are the simplest form of rank ordering. Bradley & El-Helbawy (1976) 
present the results of a consumer preference test conducted by General Foods Corporation. 
Brew strength, roast color and coffee brand were each tested at two levels, resulting in 8 
different coifee treatments. Each treatment was paired with the others for a paired test, resulting 
in 8x7=56 pairs of cups of coffee. In the test itself, 56 pairs of cups of coffee were tasted by 
each of 26 consumers. 



Preferred 


Treatment not preferred 






Treatment 


SDY 


SDX 


SLY 


SLX 


WDY 


WDX 


WLY 


WLX 


SDY 


mm 


15 


Ih 


16 


19 


14 


19 


16 


SDX 


11 


mm 


10 


15 


15 


14 


15 


12 


SLY 


11 


16 




15 


15 


14 


18 


15 


SLX 


10 


11 


11 




14 


11 


15 


13 


TOY 


7 


11 


11 


12 




9 


14 


13 


WDX 


12 


12 


12 


15 


17 




16 


18 


HLY 


7 


11 


8 


11 


12 


10 




12 


WLX 


10 


14 


11 


13 


13 


8 


14 





Table 1 . Coffee Preferences of 26 consumers. 



The preference data are presented in Table 1. Here are shown counts of the number of 
consumers who preferred the row treatment to the column treatment. Each treatment is 
conceptualized to represent the additive effect of three facets: brew strength (S or W), roast 
color (D or L) and brand (X or Y). Thus each facet contains two elements. The elements were 
not identified ir the published data, but have been assigned convenient labels for use here (e.g. 
S = Strong). The paired comparison is specified by opposing the measures of the three facet 
elements of the second treatment against those of the first. 



Measurement Model 



109 i * - wj ' ( B "* w - 

where P^^. is the probability that combination brt is preferred to b Yt* 
B> is the brew strength of element b 
R, is the roast color intensity of dement r 
T, is the brand type measure of brand t 
By is the brew strength of element b' 
R,- is the roast color intensity of element r* 
TV is the brand type measure of brand t* 

Appendix 1 contains the Facets (Linacre 1988) specifications for this analysis. There are 3 
facets: Brew, Roast and Brand. The ordering of the facets in the data records is specified in the 
"Entered statement Each data line (following "Data=") consists of element numbers for the 
first treatment in order by facet, followed by the element numbers for the second treatment in 
the same order. Finally, the number of times the first treatment is preferred over the second 
is recorded. Thus "1,2,1,2,1,2,14" means that Treatment 1 is "1,2,1", i.e, facet 1 element 1, 
facet 2 element 2, facet 3 element 1, which is Strong, Light, Brand Y (SLY). Similarly, 
Treatment 2 is Weak, Dark, Brand X (WDX). Finally SLY is preferred to WDX fourteen 
times. To assist with interpretation, greater preference (i.e. greater scores) correspond to more 
positive measures in all three facets. This is the meaning of "Positive= 1,2,3". The frame of 
reference is established by anchoring the measures of the least preferred elements at zero. 
Consequently, all three facets are non-centered. 

The "Models*" statement is "Models«?,?,?,-?,-?,-?,B26". This specifies that, from the sum 
of the measures corresponding to the first set of three elements, the sum of the measures of the 
second set of three elements is to be subtracted. From the resulting logit value the number of 
expected successes in 26 binomial trials (B26) is to be estimated. 

The " Labels =" section identifies the names of the facets and the elements within the facets. 

The results of the Facets analysis are presented graphically in Figure 1. The measures 
themselves are presented in Table 2. The Brew and Roast elements are noticeably different. 
The Brands are almost indistinguishable. Measures preceded by "A" are preset to establish the 
frame of reference. The count is that of the number of cells in which each element is contrasted 
with the other element in the same facet The Observed Average, e.g. 14.9, is the average 
number of times a treatment containing that element, e.g. "Strong" is preferred over a treatment 
containing the other element, e.g. "Weak" in the same facet. 
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Table 2. Measures from Coffee Preference paired comparisons. 
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Figure 1. Depiction of measures from Coffee Preference data. 

Paired Comparisons; Ties Allowed 

Allowing subjects the option of "no difference" complicates the analysis of paired comparisons 
(Davidson 1970). The approach used here is analogous to a rating scale. When treatments A 
and B are compared, a preference of A is rated with 2, a preference of B with 0, and a tie is 
rated 1. Springall (1973) presents such data. 28 Assessors were asked to state which treatment 
of a pair had the greater flavor strength. Three flavor concentrations were crossed with three 
gel concentrations giving 9 different treatments. There were thus 9x8 pairings. The data is 
shown in Table 3. The numbers give the count of assessors who stated that the row treatment 
the stronger flavor. The numbers after the comma are counts of those who perceived "no 
difference". 
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Table 3. Flavor strength comparisons by about 28 assessors. 



Measurement Model 



log ( jVrt ) - (S,+G ff ) - (S^*^) - Fj 

where P,^ is the probability that gel sg is rated relative to gel s'g' in category j 
S, is the flavor concentration strength element s 
G g is the gel concentration of element g 
$.. is the flavor concentration strength of element s* 
G g . is the gel concentration of dement g* 
Fj is the additional strength required to be rated in category j, j=0,2 

Appendix 2 contains the Facets specifications for this analysis. The chief additional feature is 
that the observation model is no longer binomial trials, but categories of the three category rating 
scale. The frame of reference is established by anchoring the lowest gel concentration at 0 
logits. The results of this analysis are shown in Figure 2 and Table 4. 
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Table 4. Measures obtained from Coffee Preference Data 
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Figure 2. Depiction of Measures from Flavor Strength Comparisons 
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Rank ordering 



A convenient method of constructing measures from rank orders is to regard the rankings as 
ordered categories on a rating scale. The scale definition is established spontaneously by each 
judge. With this approach, tied and partial rankings present no unusual difficulties. Critchlow 
(1985 p. 119) presents the partial rankings of five types of crackers by 22 small boys and also 
by 16 mothers. He reports rankings of only their top 3 chokes. The aim of the analysis is to 
compare how the boys ranked the crackers with how the mothers did. 



Measurement Model 




where is the probability that cracker c is ranked in category j 
C e is desirability of Cracker c 

Fj is the additional desirability required to be ranked in category j, j =0,3 
Note: ranks are converted into rating as follows: 
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The data it presented in Table 5. The specifications for a BIGSTEPS (Wright et al. 1992) 
analysis of this data are shown in Appendix 3. Each set of three letters in the data corresponds 
to one ranking. Three crackers in each ranking are assigned their rank order number. The 
unranked, but less preferred crackers, are given the joint rank of 4. Hie results of the two 
BIGSTEPS analyses, one for the boys and the other for the mothers, are shown in Table 6. 
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Partial rankings are in the form: first, second, third choice. A=animal crackers, 
C= cheese crackers, G= graham crackers, R=Ritz crackers, S=saltines. 



Table 5. Partial ranking data of cracker preference. 
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Table 6. Measures from Boys and Mothers partial rankings. 
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Figure 3. Comparison of Boys and Mothers crackers preference measures. 



The measurement analysis has provided us with two frames of reference: that of the boys and 
that of the mothers. They are compared in Figure 3. We can now determine how well each 
mother's ranking fits in the boys' frame of reference and how well each boy's ranking fits in 
the mothers* frame of reference. Two further analyses were performed in which all the boys* 
and mothers' rankings were included In both analyses. In the first analysis, the calibrations for 
the crackers were anchored at the values obtained from the earlier boys* analysis. In the second 
analysis, the calibrations were anchored at the values obtained from the earlier mothers' 
analyses. Thus two fit statistics were obtained for each ranking - one in the mother's frame of 
reference, one in the boy's frame of reference. The fit statistic used is a mean-square variance 
ratio statistic with expectation 1, minimum value 0, and infinite maximum value. 

Figure 4 is a cross-plot of the two values of the ratio-scale fit statistic obtained for each ranking. 
As can be seen, it is clear that most rankings fit with their own frame of reference, but misfit 
the other frame. There are exceptions, those marked by arrows and those in the top right 
quadrant, which could provoke further investigation. 
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Figure 4. Diagnostic plot of Boys and Mothers rankings of crackers. 



Conclusion 

In some circumstances, judge-assigned rankings have distinct advantages over judge-assigned 
ratings. This paper demonstrates several methods of producing measures from rankings, and 
also illustrates their value in enhancing our understanding of the latent variables, and also 
identifying features of the data which depart from the consensus. 
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Appendix 1. Coffee Preference Data: Facets specifications. 
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(SDY vs. SDX: 15 out of 26 preferred SDY) 



Appendix 2, Flavor Strength Data: Facets specifications. 

Title*Flavor Strength using Rating Scale 

Facets»2 

Positive-1,2 

Non-center* 1 , 2 

Entered~l, 2,1,2 
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| (More data follows) 




Appendix 3. Partial rankings of Crackers: BIGSTEPS specifications. 
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Graham Craekara 4133244423241242434233 4344443144433414 
Rita Craekara 4444424342442444241444 2244234334114344 
Saltinet 3444442144434433444444 4122111412241131 
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